Let’s start with some background about patchmgr: “patchmgr” is a tool specifically designed for managing patching and updates on Oracle Exadata Database Machine systems. Exadata is Oracle’s engineered system designed to deliver high performance, availability, and scalability for Oracle Database workloads.
The Exadata Patch Manager Update Utility streamlines the process of applying patches, updates, and firmware upgrades to various components of the Exadata infrastructure, including database servers, storage servers, networking components, and other hardware and software elements.
In our case, the objective is to patch all the database nodes to 21.2.18.0.0.221111.1. After starting the patchmgr the process halted/failed at backup before upgrade/update of the patch.
01:34:25 root@ORA-X1:+ASM1 % ./patchmgr -dbnodes ~oracle/dbs_group_909-10 -backup -iso_repo /oracle_workspace/Frdz/ExaUpgrade/ESS/p34694885_212000_Linux-x86-64.zip -target_version 21.2.18.0.0.221111.1 -allow_active_network_mounts
************************************************************************************************************
NOTE patchmgr release: 23.240129 (always check MOS 1553103.1 for the latest release of dbserver.patch.zip)
NOTE
WARNING Do not interrupt the patchmgr session.
WARNING Do not resize the screen. It may disturb the screen layout.
WARNING Do not reboot database nodes during update or rollback.
WARNING Do not open logfiles in write mode and do not try to alter them.
************************************************************************************************************
2024-03-12 01:35:09 +0900 :Working: Verify SSH equivalence for the root user to node(s)
2024-03-12 01:35:09 +0900 :WARNING: patchmgr is launched from a remote/shared filesystem
2024-03-12 01:35:09 +0900 :WARNING: Ensure the filesystem is not hosted from any of the target nodes listed
2024-03-12 01:35:10 +0900 :INFO : SSH equivalency verified to host ORA-X1
2024-03-12 01:35:11 +0900 :INFO : SSH equivalency verified to host ORA-X2
2024-03-12 01:35:11 +0900 :SUCCESS: Verify SSH equivalence for the root user to node(s)
2024-03-12 01:38:46 +0900 :Working: Initiate backup on 2 node(s).
2024-03-12 01:38:46 +0900 :Working: Check for enough free space on node(s) to transfer and unzip files.
2024-03-12 01:38:54 +0900 :SUCCESS: Check for enough free space on node(s) to transfer and unzip files.
2024-03-12 01:38:55 +0900 :INFO : Preparing nodes: ORA-X1 ORA-X2
^@2024-03-12 01:40:20 +0900 :Working: dbnodeupdate.sh running a backup on node(s).
2024-03-12 01:41:51 +0900 :ERROR : dbnodeupdate.sh backup failed on one or more nodes
SUMMARY OF ERRORS FOR ORA-X1:
ORA-X1: ERROR: Inactive lvm (/dev/mapper/VGExaDb-LVDbSys2) (15G) not equal to active lvm /dev/mapper/VGExaDb-LVDbSys1 (45G). Backups will fail. Re-create it with proper size
2024-03-12 01:41:52 +0900 :ERROR : dbnodeupdate.sh backup on ORA-X1
flock: 200: No locks available
The above error is very clear: LV which holds the Primary ISO image, should have an equal counter LV with size to hold the backup so that it will be used to rollback the upgrade patch version if the latest upgrade fails
% lvscan
ACTIVE '/dev/VGExaDb/LVDbTmp' [30.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbVarLog' [18.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbSys2' [15.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbVar1' [5.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbSwap1' [16.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbHome' [100.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbSys1' [45.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbVar2' [2.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbOra1' [300.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDoNotRemoveOrUse' [2.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbVarLogAudit' [1.00 GiB] inherit
ISO IMAGE HOLDER : /dev/VGExaDb/LVDbSys1 must be equal to /dev/VGExaDb/LVDbSys2 in size to take backup
lvm lvextend -L+30G --verbose /dev/mapper/VGExaDb-LVDbSys2
Archiving volume group "VGExaDb" metadata (seqno 17).
Extending logical volume VGExaDb/LVDbSys2 to 45.00 GiB
Size of logical volume VGExaDb/LVDbSys2 changed from 15.00 GiB (3840 extents) to 45.00 GiB (11520 extents).
Loading table for VGExaDb-LVDbSys2 (252:2).
Suspending VGExaDb-LVDbSys2 (252:2) with device flush
Resuming VGExaDb-LVDbSys2 (252:2).
Creating volume group backup "/etc/lvm/backup/VGExaDb" (seqno 18).
Logical volume VGExaDb/LVDbSys2 successfully resized.
[root@ORA-X1 dbserver_patch_240129]# lvscan
ACTIVE '/dev/VGExaDb/LVDbTmp' [30.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbVarLog' [18.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbSys2' [45.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbVar1' [5.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbSwap1' [16.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbHome' [100.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbSys1' [45.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbVar2' [2.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbOra1' [300.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDoNotRemoveOrUse' [2.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbVarLogAudit' [1.00 GiB] inherit
# mkfs -t ext3 -b 4096 /dev/mapper/VGExaDb-LVDbSys2
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=256 blocks, Stripe width=256 blocks
2949120 inodes, 11796480 blocks
589824 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
360 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
# lvscan
ACTIVE '/dev/VGExaDb/LVDbTmp' [30.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbVarLog' [18.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbSys2' [45.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbVar1' [5.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbSwap1' [16.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbHome' [100.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbSys1' [45.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbVar2' [2.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbOra1' [300.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDoNotRemoveOrUse' [2.00 GiB] inherit
ACTIVE '/dev/VGExaDb/LVDbVarLogAudit' [1.00 GiB] inheritAfter extending the LV, the patchmgr could perform a backup successfully, and patching was completed without any issues.
Reference: Exadata YUM Pre-Checks Fails with ERROR: Inactive lvm not equal to active lvm. Backups will fail. (Doc ID 1988429.1)
